Interlaced bi-sensor super-resolution enhancement

ABSTRACT

Interlaced bi-sensor super-resolution enhancement techniques and a resultant scalable pixel array suitable for a mega-pixel design are disclosed. The method includes interlacing a first array of pixels of a first size with a second array of pixels of a second size. The interlacing of the first array of pixels with the second array of pixels avoids crossing two or more photosensitive areas of the first array of pixels and the second array of pixels.

FIELD OF THE INVENTION

The invention relates to enhancement of images and, more particularly,to interlaced bi-sensor super-resolution enhancement techniques and aresultant scalable pixel array suitable for a mega-pixel design.

BACKGROUND OF THE INVENTION

Enhancing the resolution of an image of a camera can be important formany different applications. For example, enhancing resolution of animage can find application in surveillance, military and defense,aerospace, traffic control, medical imaging, and commercial applicationssuch as smart phones that do not have the luxury of large sensor size.

One approach of enhancing a resolution of an image is through softwarealgorithms. For example, a common way to enhance a resolution is throughsuper-resolution techniques depending on software post-processingalgorithms. These software post-processing algorithms can be applicableto multi-framed or single-frame, where the effectiveness depends on thescene.

Another approach may be to use two sensors (bi-sensor super resolution);however, there is an increased cost associated with this method. Theincreased cost is not only due to the need of two sensors, but also dueto needing two full cameras, which in turn, requires two lenses (large,bulky, expensive), two processors, two batteries, two separate outputstreams, etc. Additionally, a bi-sensor approach also requires ensuringidentical optical paths for both sensors, which requires the use of abeamsplitter and a calibrated optical setup. This significantlyincreases complexity.

SUMMARY OF THE INVENTION

In an aspect of the invention, a method comprises interlacing a firstarray of pixels of a first size with a second array of pixels of asecond size. The interlacing of the first array of pixels with thesecond array of pixels avoids crossing two or more photosensitive areasof the first array of pixels and the second array of pixels.

In an aspect of the invention, a method comprises: selecting a firstarray of pixels comprising pixels of a first pitch; selecting a secondarray of pixels comprising pixels of a second pitch which is differentthan the first pitch; and interlacing the first array of pixels and thesecond array of pixels to form a repeatable modular block for aninterlaced bi-sensor super-resolution pixel array while avoidingcrossing of photosensitive areas of the first array of pixels and thesecond array of pixels.

In an aspect of the invention, a repeatable modular block of interlacedbi-sensor super-resolution pixel array comprises a first array of pixelscomprising pixels of a first pitch interlaced with a second array ofpixels comprising pixels of a second pitch which is different than thefirst pitch such that photosensitive areas of the first array of pixelsand the second array of pixels are minimized.

In an aspect of the invention, a computer system for interlacing pixelarrays, comprises: a CPU, a computer readable memory and a computerreadable storage media; program instructions to select a first array ofpixels comprising pixels of a first pitch; program instructions toselect a second array of pixels comprising pixels of a second pitchwhich is different than the first pitch; and program instructions tointerlace the first array of pixels and the second array of pixels toform a repeatable modular block for an interlaced bi-sensorsuper-resolution pixel array while avoiding crossing of photosensitiveareas of the first array of pixels and the second array of pixels;wherein the program instructions are stored on the computer readablestorage media for execution by the CPU via the computer readable memory.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described in the detailed description whichfollows, in reference to the noted plurality of drawings by way ofnon-limiting examples of exemplary embodiments of the present invention.

FIG. 1 shows two pixel arrays which can be combined into a dual arraysingle sensor;

FIG. 2 represents an interlacing of the two pixel arrays shown in FIG.1;

FIG. 3 shows two pixel arrays which can be combined into a dual arraysingle sensor;

FIG. 4 represents an interlacing of the two pixel arrays shown in FIG.3;

FIG. 5 shows two pixel arrays which can be combined into a dual arraysingle sensor;

FIG. 6 represents an interlacing of the two pixel arrays shown in FIG.5;

FIG. 7 shows two pixel arrays which can be combined into a dual arraysingle sensor;

FIG. 8 represents an interlacing of the two pixel arrays shown in FIG.7; and

FIG. 9 shows a computing infrastructure to implement aspects of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

The invention relates to enhancement of images and, more particularly,to interlaced bi-sensor super-resolution enhancement techniques and aresultant scalable sensor array suitable for a mega-pixel design. Forexample, the present invention provides a method to form building blocksof pixel arrays for copy and paste operations to form a pixel sensorarray suitable for a mega-pixel design. In this way, the invention canbe implemented in an interlaced bi-sensor super-resolution camera.

More specifically, the present invention provides a dual-array, singleimage sensor with different pitch interlaced pixels, especially used forsuper resolution (SR) of images. Advantageously, the method of thepresent invention satisfies the sampling rate diversity requirement inonly one single sensor; compared to the need for two sensors to satisfyits different sampling rate requirement of conventional methods andsystems.

In embodiments, the sensor design interlaces two image sensor arrayswithin one sensor. The two interlaced arrays have a different pitch thatwill allow this dual array single sensor to function as a bi-sensorsuper-resolution camera. The array has pixels that are laid out in amodular (repeatable) block that will allow for typical image sensorarray layout and which provides the capabilities of copy and pasteoperations to build a scalable pixel array sensor suitable formega-pixel design. In this way, the present invention advantageouslyresults in a very simplified sensor that cost the same as a singlecamera single image sensor setup and has no major sacrifice to siliconarea.

The sensor design of the present invention can be manufactured in anumber of ways using a number of different tools. In general, though,the methodologies and tools are used to form structures with dimensionsin the micrometer and nanometer scale. The methodologies, i.e.,technologies, employed to manufacture the sensor design of the presentinvention have been adopted from integrated circuit (IC) technology. Forexample, the structures of the present invention are built on wafers andare realized in films of material patterned by photolithographicprocesses on the top of a wafer. In particular, the fabrication of thesensor design of the present invention uses three basic building blocks:(i) deposition of thin films of material on a substrate, (ii) applying apatterned mask on top of the films by photolithographic imaging, and(iii) etching the films selectively to the mask. The layout of thepresent invention is applicable to different technologies including, forexample, CMOS 180 nm technology from Global Foundries as well as LF 150nm technology, by KACST.

FIG. 1 shows two arrays, Array A and Array B which can be combined intoa dual array single sensor to function as bi-sensor super-resolutioncamera or building blocks as shown in FIG. 2. In the non-limitingillustrative example of FIG. 1, Array B is shown in a block of 4×4,while Array A is shown in a block of 5×5; although other block sizes arecontemplated by the present invention. The pitch of Array A is labeledP₁ and the pitch of Array B is labeled P₂.

In embodiments, Array A has pixels (Pixel A) of a smaller area (smallerpitch in both dimensions horizontal and vertical) with a specific ratiowhen compared to Array B (that has Pixel B). In this example, Pixel Bhas a pitch that is 1.25 times that of Pixel A, therefore, in theexample of FIG. 1, Pixel A has a pitch of 8 μm and Pixel B has a pitchof 10 μm, which is a ratio of 1.25. This case is only specific to theexample ratio of 1.25, where other ratios may require different blocksizes. It should be understood by those of skill in the art, that otherpitches or different block sizes are thus contemplated by the presentinvention.

As further noted herein, it should be understood by those of skill inthe art that the pixels contain a photodiode that it is usually smallerthan the size of the pixel to leave room for the electronics requiredfor pixel conditioning and control (e.g., such as the 3-transistoractive pixel sensor design). The room for electronics is necessary inalmost all CMOS image sensor design, except for the case of back-sideillumination technology, where the electronics are integrated in adifferent layer than the photodiode. For these reasons, the presentinvention contemplates the need for different pitches for the differentarrays in order to prevent blockage (e.g., crossover) of photosensitiveareas as described herein when interlaced as shown in FIG. 2.

Also, in the example of FIG. 1, the pixels and photodiodes are assumedto be squares; although other shapes and sizes are contemplated by thepresent invention. Finally, the areas of the photodiodes are assumed tobe equal (a₁xa₁=a₂xa₂) to maintain similar dynamic range, sensitivityand responsively characteristics between the two arrays.

FIG. 2 represents both of the arrays (Array A and Array B) overlapped inan interlaced manner without crossing photosensitive areas in accordancewith aspects of the present invention, e.g., dual array single sensor.The dual array single sensor is more specifically representative of arepeatable, scalable modular block which can be used for a dual arraysingle sensor suitable for a mega-pixel design.

As should be understood, a cross in two or more photosensitive areas mayresult in a variation of the pixel pitch. In FIG. 2, for example, atpixel (0,0), the first pixel on the top left corner, both photodiodesoverlap fully, meaning that only one photodiode laid out in thatlocation can serve for both pixels. Also, some of the photodiodes maycome in close proximity of each other; although, this is not an issue inFIG. 2. However, it may be an issue in some layouts as there is aminimum well spacing between the photodiodes to avoid design rule check(DRC) violation. Accordingly, in implementing the present invention, adetermination is made as to the minimum well spacing between thephotodiodes to avoid design rule check (DRC) violation.

Using the infrastructure of FIG. 6, the condition to avoid overlapbetween square or rectangular photodiodes to result in the interlacedarray of FIG. 2 can be derived as follows:

P₂>P₁   (1)

P ₂ =s×P ₁   (2)

s=scaling factor (in this case 1.25)   (3)

a=P ₂ −P ₁−min_spacing_rule   (4).

It should be understood that “a” represents the size of the photodiodeand a² is thus representative of the area of the photodiode. It shouldbe further understood that the present invention is not limited to ascaling factor of 1.25, and that other scaling factors can beimplemented with the present invention, based on size of the pixels orarrays and other considerations described herein, e.g., avoidance ofcrossing any of the photosensitive areas (blockage of photosensitiveareas when arrays are overlapped). For example, the scaling factor canbe based on (i+1)/i, where i is representative of an array, e.g., a 4×4array will result in (4+1)/4 which equals 1.25 scaling factor. Inembodiments, an upscaling factor (e.g., “s”) can be based on a 1megabyte pixel in, e.g., a 6×6 array (36 megabyte array), where thescaling factor can be, e.g., (6+1)/6 which equals 1.17.

For the case shown in the 5×5 block of FIG. 1, with pixel pitch of 8 μmand 10 μm, and a minimum well spacing rule of 0.1 (with all units inmicrometers), as an example:

P ₂=1.25×P ₁   (5)

a=1.9   (6)

The minimum well spacing rule is technology dependent and, as such, theminimum well spacing rule of 0.1 is provided as an illustrative example.The fill-factor (FF), which is a ratio of photosensitive part of thepixel to entire pixel area, can be estimated depending on the number ofphotosensitive parts of the pixel, which varies between 2 to 4, asfollows:

2a ² /P ₂ ² <FF<4a ² /P ₂ ²   (7).

It should be understood that the interlaced array 100 of FIG. 2 can beused as a building block for a dual array single sensor. For example,the building block of FIG. 2 can be used in copy and paste designprocesses known to those of skill in the art to complete a scalablearray suitable for a mega-pixel design. For example, the presentinvention can be provided in design processes which preferably employand incorporate hardware and/or software modules for synthesizing,translating, or otherwise processing a design/simulation functionalequivalent of the components, circuits, devices, or logic structuresshown in FIG. 2 to generate a netlist which may contain equivalentdesign structures of the array of FIG. 2. Netlist may comprise, forexample, compiled or otherwise processed data structures representing alist of wires, discrete components, logic gates, control circuits, I/Odevices, models, etc. that describes the connections to other elementsand circuits in an integrated circuit design of FIGS. 1 and 2. Resultingdesign structures, representative of the arrays of FIGS. 1 and 2, canreside on a storage medium or programmable gate array in a data formatused for the exchange of data of mechanical devices and structures (e.g.IGES, DXF, Parasolid XT, JT, DRG, GDSII (GDS2), GL1, OASIS, etc.). Thedesign structure when processed by an ECAD system generates a logicallyor otherwise functionally equivalent form of one or more of theembodiments of the invention shown in FIG. 2 (and FIGS. 4, 6 and 8).

FIGS. 3 and 4 provide alternative processes to provide an interlacedarray in accordance with aspects of the present invention. In FIGS. 3and 4, though, super resolution can be performed using only onedimensional dual pitch, instead of two. That is, FIG. 3 shows a scalingin one dimension, e.g., horizontal (but not both horizontal andvertical). In this case, the FF can be increased significantly. Also inthis case, the repeatable modular block will be smaller, as shown inFIG. 4 (compared to FIG. 2).

More specifically, FIG. 3 shows two arrays, Array A and Array B whichcan be combined into a dual array single sensor to function as abi-sensor super-resolution camera as shown in FIG. 4. More specifically,the interlaced array 100′ of FIG. 4 is representative of a repeatable,scalable modular block which can be used to for a dual array singlesensor suitable for a mega-pixel design.

In the non-limiting illustrative example of FIG. 3, Array B is shown ina block of 4×4, while Array A is shown in a block of 5×5. The pitch ofArray A is P₁ (horizontal) and P₂ (vertical), whereas, the pitch ofArray B is P₂ for both horizontal and vertical. Hence, Array A hasrectangular pixels and Array B has square pitches. In embodiments, ArrayA has pixels (Pixel A) of a smaller area (smaller pitch in thehorizontal) with a specific ratio when compared to Array B (that hasPixel B). Finally, the areas of the photodiodes are assumed to be equal(a₁xa₁=a₂xa₂) to maintain similar dynamic range, sensitivity andresponsively characteristics between the two arrays.

As shown in FIG. 4, using the equations above it is now possible toobtain a modular, repeatable, scalable modular block 100′ of 5×2 primarypixels, as a minimum building block. As in the representation of FIG. 2,the modular, repeatable building block of FIG. 4 can be used for a dualarray single sensor suitable for a mega-pixel design. Although not aseffective as the building block shown in FIG. 2, the building block ofFIG. 4 will provide enhanced resolution. Also, it should be understoodby those of skill in the art that an upscale factor need only be used ina single dimension in this implementation which, accordingly, will onlyincrease the bit size in one dimension. Similar to that explained withregard to FIG. 2, the building blocks of FIG. 4 can be used asrepeatable, scalable modular blocks 100′ suitable for a mega-pixeldesign. For example, the building block of FIG. 2 can be used in copyand paste design processes known to those of skill in the art tocomplete a dual array single sensor.

As shown in FIGS. 5 and 6, using a variable sampling rate for theprimary data set (Array A) can help increase the FF. For example, inFIG. 5, Array B has a pitch P₂ (10 units) that is larger than Array A bya factor of 1.25. The photodiodes in this case are larger than thecondition derived in equation 6, which results in some of thephotodiodes in the 4×4 block overlapping. However, as shown in FIG. 6,some of the photodiodes of the Array A are removed in order to avoid DRCerrors or to avoid modifying the pitch of Array B. The result of theinterlaced layout 100″ is shown in FIG. 6, where Array A samples are 36%less than the original case for Array A or 20% of both arrays together.This was tested and shown not to have any significant impact on theenhanced images, in that known software algorithms can extrapolate forthe missing photodiodes, while still providing enhanced imagery.

As shown in FIG. 7, a variable sampling rate can also be implemented byremoving pixels from secondary array (Array B), as shown in FIG. 8. Ashown in the interlaced layout 100″' of FIG. 8, Array B samples are 50%less than the original case, also only 20% less than both arraystogether. This was also tested and shown not to have any significantimpact on the enhanced images. It should be understood by those of skillin the art that any combination of pixels can be removed from theprimary array (Array A) or the secondary array (Array B) in order toavoid crossover of the pixels, as described herein.

The present invention may be embodied as a system, method or computerprogram product. The present invention may take the form of a hardwareembodiment, a software embodiment or a combination of software andhardware. Furthermore, the present invention may take the form of acomputer program product (program instructions) embodied in any tangiblestorage medium of expression having computer-usable program codeembodied in the medium, which implements the methods, techniques andprocesses herein. The computer readable storage media may be any mediumthat can contain, store, or communicate the program for use by or inconnection with the instruction execution system, apparatus, or device.The computer readable storage media is not a signal per se, ortransitory. The computer-usable or computer-readable medium may be, forexample, an electronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, device, or propagation medium.

FIG. 9 shows an illustrative environment 10 for managing the processesin accordance with the invention. The environment 10 includes a serveror other computing system 12 that can perform the processes describedherein. The server 12 includes a computing device 14 which can beresident on a network infrastructure or computing device. The computingdevice 14 includes a processor (CPU) 20, memory 22A, an I/O interface24, and a bus 26. In addition, the computing device includes randomaccess memory (RAM), a read-only memory (ROM), and an operating system(O/S). The computing device 14 is in communication with the external I/Odevice/resource 28 and the storage system 22B. The I/O device 28 cancomprise any device that enables an individual to interact with thecomputing device 14 (e.g., user interface) or any device that enablesthe computing device 14 to communicate with one or more other computingdevices using any type of communications link. The processor 20 executescomputer program code (e.g., program control or program instructions44), which can be stored in the memory 22A and/or storage system 22B.While executing the computer program code, the processor 20 can readand/or write data to/from memory 22A, storage system 22B, and/or I/Ointerface 24. The program code executes the processes of the invention.

In embodiments, the present invention comprises the CPU, computerreadable memory and computer readable storage media. In particularembodiments, program instructions are configured to select a first arrayof pixels comprising pixels of a first pitch. Program instructions areconfigured to select a second array of pixels comprising pixels of asecond pitch which is different than the first pitch. Programinstructions are configured to interlace the first array of pixels andthe second array of pixels to form a repeatable modular block for aninterlaced bi-sensor super-resolution pixel array while avoidingcrossing of photosensitive areas of the first array of pixels and thesecond array of pixels. Program instructions are configured to providethe other methods, steps and techniques already described herein. Theprogram instructions are stored on the computer readable storage mediafor execution by the CPU via the computer readable memory.

The foregoing examples have been provided for the purpose of explanationand should not be construed as limiting the present invention. While thepresent invention has been described with reference to an exemplaryembodiment, Changes may be made, within the purview of the appendedclaims, without departing from the scope and spirit of the presentinvention in its aspects. Also, although the present invention has beendescribed herein with reference to particular materials and embodiments,the present invention is not intended to be limited to the particularsdisclosed herein; rather, the present invention extends to allfunctionally equivalent structures, methods and uses, such as are withinthe scope of the appended claims.

What is claimed is:
 1. A method comprising interlacing a first array ofpixels of a first size with a second array of pixels of a second size,wherein the interlacing of the first array of pixels with the secondarray of pixels avoids crossing two or more photosensitive areas of thefirst array of pixels and the second array of pixels, and forms arepeatable modular block for an interlaced bi-sensor super-resolutionpixel array.
 2. The method of claim 1, wherein the repeatable modularblock is copied and pasted to form a scalable array for a mega-pixeldesign.
 3. The method of claim 2, wherein the first array of pixels andthe second array of pixels comprise a different pitch.
 4. The method ofclaim 2, wherein super resolution is performed using only onedimensional dual pitch, instead of two.
 5. The method of claim 4,wherein scaling is provided in one dimension.
 6. The method of claim 5,wherein the scaling is provided in a horizontal direction.
 7. The methodof claim 1, wherein the first array of pixels have a first pitch and asecond pitch.
 8. The method of claim 7, wherein the first pitch ishorizontal and the second pitch is vertical.
 9. The method of claim 8,wherein the first array of pixels have a pitch that is both horizontaland vertical.
 10. The method of claim 9, wherein the first array ofpixels has rectangular pixels and the second array of pixels has squarepitches.
 11. The method of claim 10, wherein first array of pixels haspixels of a smaller area with a specific ratio when compared to that ofthe second array of pixels.
 12. The method of claim 11, wherein thefirst array of pixels have a smaller pitch in a horizontal directionthan the pixels of the second array of pixels.
 13. The method of claim12, wherein areas of photodiodes of the first array of pixels and thesecond array of pixels are equal to maintain similar dynamic range,sensitivity and responsively characteristics between the two arrays. 14.The method of claim 1, further comprising using an upscale factor onlyin a single dimension which will only increase the bit size in onedimension.